Evaluate Train Phrases

The training model includes ALL training phrases defined on all flows associated with the respective bot. The Train Set is the actual sample data used to train the bot’s NLP model. The model uses the data to see and learn.

The Train Set feature (Train Set tab) enables you to:

Evaluate the bot training model

Note:  The evaluation metrics are using trained data and there might be new settings not included. Check the last train date and train the bot model by clicking the Train button.

The Train Set page provides you with information from the TrainLog file (with matching percentage for each training phrase). You can check the general metrics of the bot training model and the general metrics per flow.

Test a train phrase (utterance)

To test a phrase, in the Test field, enter the phase, click on the Test button and select the language:

If testing a phrase (intent) returns no matches found, you can further fine-tune the model using  NLP and NER settings.

View utterance evaluation details

To view the evaluation details of a flow, in the table, click on the desired flow. Below the flow the utterances are listed together with their matching status, the top two matched flows (if DRUID matched the utterance on multiple flows or if the utterance was matched to a flow (or flows) other than the expected one) and their matching score.

The color of the intents provides you with valuable information, as follows:

  • Black. The system matches the utterance (training phrase) on a single, corresponding flow. However, you should check the score to see if the phrase is strong enough.
  • Red. The system matches that utterance with other flow than the one it is defined on (wrong match).
  • Orange. The system matches the utterance (training phrase) on multiple flows.

If you want to edit an utterance click the dots displayed inline with the utterance and click Edit.

The Add to Train Set page appears that allows you to edit the utterance. You can also improve the train set by editing / deleting training phrases from the expected flow or from the top matched flows.

Edit the desired training phrase and click the Train button to train the bot with the new training phrase.

Hint:   If you selected Enable auto train on the bot (bot details page, NLP section, there is no need to click the Train button.

Evaluate child models

For DRUID version 1.73 and higher you can evaluate the NLP sub-model of parent flows. To do so, identify the parent flow whose NLP sub-model you want to evaluate and in the Child Intent column, click the Child model icon. In line with the parent flow, two labels are displayed: Error (in red) and Accuracy (in green).

The NLP model of the parent flow appears.

Note:  The NLP parameters set on the bot apply to the bot model and to child models.

You can evaluate child models similar to evaluating bot training phrases:

  • Test the child model understanding.
  • View the evaluation details of a specific train phrase.
  • Understand why the NLP sub-model associated an utterance to a specific child intent.
  • Add to Test set.

Important!  For child intents, if the TargetMatchScore (the target threshold for matching utterance) is not reached, DRUID does not trigger IntentNotRecognized.

To go back to the bot model, at the top-left side of the Evaluation page (above the test child model understanding area), click Back to Bot Model.

Download bot train model logs

  • Download – Downloads the bot training model in Excel format. It is the training model visualized in the Train Set page.  For information of the metrics you can find in the train set log, see Train Status.
  • DownloadTrainLog - Downloads (in your default Download folder) a JSON file, which contains comprehensive training phrases data: status per training phrases, score for each utterance, etc.
  • DownloadTrainErrors – Downloads (in your default Download folder) a JSON file, which contains the errors occurred (if any) during the last NLP model training.
Related Topics Link IconRelated Topics